| Kafka topic分区增加副本 | 您所在的位置：网站首页 › kafka topic增加分区 › Kafka topic分区增加副本 |
|
Kafka中topic的每个分区可以设置多个副本。如果副本数为1，当该分区副本的leader节点宕机后，会导致该分区不可用。故需要设置多副本来保证可用性。 实际项目中，存在项目初期创建了副本数为1的topic，但是后期又需要扩大副本数的场景。通常不能直接删除topic重建，可以通过如下操作实现。 准备工作创建副本为1的topic kafka-topics --zookeeper mdw:2181/kafka --create --replication-factor 1 --partitions 3 --topic test_topic查看topic信息 kafka-topics --describe --zookeeper mdw:2181/kafka --topic test_topic 输出： Topic:test_topic PartitionCount:3 ReplicationFactor:1 Configs: Topic: test_topic Partition: 0 Leader: 364 Replicas: 364 Isr: 364 Topic: test_topic Partition: 1 Leader: 365 Replicas: 365 Isr: 365 Topic: test_topic Partition: 2 Leader: 366 Replicas: 366 Isr: 366可以看出3个分区，各自都是只有一个副本。当Leader对应节点挂掉后，分区就不可用了。 增加为3副本使用官方自带的kafka-reassign-partitions.sh脚本实现。该脚本用来移动分区的副本位置。除了可以实现增加副本，还可以实现将分区的副本移动到指定机器上。该脚本的help见附录。 以下步骤实现将test_topic的各个分区增加为三副本。 制定分区及副本分配策略 创建json文件，写入如下内容： 填写说明：topic为topic名称，partition为分区号，replicas为broker id的数组 { "version": 1, "partitions": [ { "topic": "test_topic", "partition": 0, "replicas": [ 364, 365, 366 ] }, { "topic": "test_topic", "partition": 1, "replicas": [ 364, 365, 366 ] }, { "topic": "test_topic", "partition": 2, "replicas": [ 364, 365, 366 ] } ] }执行扩副本操作 本例中创建的json文件为test_topic.json 执行如下命令： kafka-reassign-partitions --zookeeper mdw:2181/kafka --reassignment-json-file test_topic.json --execute输出内容： Current partition replica assignment {"version":1,"partitions":[{"topic":"test_topic","partition":2,"replicas":[366],"log_dirs":["any"]},{"topic":"test_topic","partition":1,"replicas":[365],"log_dirs":["any"]},{"topic":"test_topic","partition":0,"replicas":[364],"log_dirs":["any"]}]} Save this to use as the --reassignment-json-file option during rollback Successfully started reassignment of partitions.这是个异步操作，执行命令后开始执行扩副本。 输出内容中有之前的副本策略，可以保存下来，用于回滚。 查看执行进度 kafka-reassign-partitions --zookeeper mdw:2181/kafka --reassignment-json-file test_topic.json --verify出现如下内容说明执行完成 Status of partition reassignment: Reassignment of partition test_topic-0 completed successfully Reassignment of partition test_topic-1 completed successfully Reassignment of partition test_topic-2 completed successfully验证是否成功 kafka-topics --describe --zookeeper mdw:2181/kafka --topic test_topic 输出： Topic:test_topic PartitionCount:3 ReplicationFactor:3 Configs: Topic: test_topic Partition: 0 Leader: 364 Replicas: 364,365,366 Isr: 364,366,365 Topic: test_topic Partition: 1 Leader: 365 Replicas: 364,365,366 Isr: 365,366,364 Topic: test_topic Partition: 2 Leader: 366 Replicas: 364,365,366 Isr: 366,365,364从输出中可以看出每个分区的副本都变成了3个。 附录# kafka-reassign-partitions This tool helps to moves topic partitions between replicas. Option Description ------ ----------- --bootstrap-server bootstrapping. REQUIRED if an absolute path of the log directory is specified for any replica in the reassignment json file --broker-list The list of brokers to which the partitions need to be reassigned in the form "0,1,2". This is required if --topics-to-move-json-file is used to generate reassignment configuration --command-config passed to Admin Client. --disable-rack-aware Disable rack aware replica assignment --execute Kick off the reassignment as specified by the --reassignment-json-file option. --generate Generate a candidate partition reassignment configuration. Note that this only generates a candidate assignment, it does not execute it. --help Print usage information. --reassignment-json-file reassignment configurationThe format to use is - {"partitions": [{"topic": "foo", "partition": 1, "replicas": [1,2,3], "log_dirs": ["dir1","dir2","dir3"] }], "version":1 } Note that "log_dirs" is optional. When it is specified, its length must equal the length of the replicas list. The value in this list can be either "any" or the absolution path of the log directory on the broker. If absolute log directory path is specified, the replica will be moved to the specified log directory on the broker. --replica-alter-log-dirs-throttle The movement of replicas between log directories on the same broker will be throttled to this value (bytes/sec). Rerunning with this option, whilst a rebalance is in progress, will alter the throttle value. The throttle rate should be at least 1 KB/s. (default: -1) --throttle The movement of partitions between brokers will be throttled to this value (bytes/sec). Rerunning with this option, whilst a rebalance is in progress, will alter the throttle value. The throttle rate should be at least 1 KB/s. (default: -1) --timeout The maximum time in ms allowed to wait for partition reassignment execution to be successfully initiated (default: 10000) --topics-to-move-json-file to move the partitions of the specified topics to the list of brokers specified by the --broker- list option. The format to use is - {"topics": [{"topic": "foo"},{"topic": "foo1"}], "version":1 } --verify Verify if the reassignment completed as specified by the --reassignment- json-file option. If there is a throttle engaged for the replicas specified, and the rebalance has completed, the throttle will be removed --zookeeper REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over. |
| CopyRight 2018-2019 实验室设备网 版权所有 |